Optimal value functions for weakly coupled systems: a posteriori 

estimates 



(N 

o 

(N 



X 



Peter Koltai and Oliver Junge* 
September 17, 2012 



Abstract 



We consider weakly coupled LQ optimal control problems and derive estimates on the sensitivity 
of the optimal value function in dependence of the coupling strength. In order to improve these 
sensitivity estimates a "coupling adapted" norm is proposed. Our main result is that if a weak 
coupling suffices to destabilize the closed loop system with the optimal feedback of the uncoupled 
system then the value function might change drastically with the coupling. As a consequence, it 
is not reasonable to expect that a weakly coupled system possesses a weakly coupled optimal value 

O function. Also, for a known result on the connection of the separation operator and the stability 
radius a new and simpler proof is given. 

•s 

1 Introduction 

When designing feedback controllers for high dimensional control systems using dynamic programming, 
psj the dimension of state space imposes natural limitations for existing methods: In general, the represen- 

tation of the optimal value function suffers from the curse of dimensionality. An efficient approximation 
is only possible if the optimal value function possesses some regularity that we can exploit. Our aim here 
is to investigate such regularity properties for systems which can be decomposed into a number of small 
^{•^ subsystems which - in a proper sense - are only weakly coupled to each other. This scenario is, e.g., 

motivated by so called networked control systems. 

More specifically, we consider a system consisting of several weakly coupled subsystems such that an 
I optimal feedback of each subsystem, i.e. without the coupling to the other systems, is easily computable 

y—( (for example, if it is an LQ system or if its state space dimension is < 3). These feedbacks together form 

an optimal feedback for the uncoupled system, the naive feedback. Not surprisingly, the naive feedback 
may fail to stabilize the coupled system for particular coupling structures even if the coupling is weak. 
In this paper, we aim to understand why and how the naive feedback fails, and how the optimal value 
function changes with increasing coupling strength. 

In particular, we consider the time-continuous linear quadratic regulator (LQR) problem, for which 
the optimal value function is a quadratic function of the form V(x) = x T Px with a symmetric positive 
definite matrix P. In Section 3 we recall some existing theory on the sensitivity analysis of the LQR 
problem. Since the norm in which the sensitivity is measured yields unsatisfactory estimates for the 
coupling of the optimal value function, in Section 4 the estimates are established in a coupling-adapted 
norm. Similar estimates are derived in Section 5 for discrete time systems. Section 6 contains a discussion 
about the obtained estimates. Finally, we turn to the question on what these estimates tell us about the 
structure of the optimal value function if we can introduce a weak coupling such that the naive feedback 
fails to stabilize the coupled system. The answer is somewhat surprising: if a coupling (no matter how 
small) destabilizes the system controlled by the naive feedback, the optimal value function for the coupled 
system differs massively from the one of the uncoupled system; cf. Section 7. 
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2 The optimal value function, uncoupled and weakly coupled 
systems 

We consider a control system 

x = f(x,u), (1) 

where the right hand side / : XxU — » R" is assumed to be C 1 and the state space X C R n , £ A 1 , and the 
set of controls U C R m , OeW, are compact. The origin of the uncontrolled system is a (possibly unstable) 
equilibrium, i.e. we have that /(0, 0) = 0. We are interested in constructing a feedback u : X — > U such 
that the origin of the closed loop system x = f(x, u(x)) is asymptotically stable. To this end, we consider 
a continuous cost function c : X x U —> [0, oo) which fulfills the condition c(x, u) = iff x = 0. Given an 
initial value xo € X and a (measurable) control function u : [0, oo) — > X, the cost along the associated 
trajectory x(t;xo,u) of (1) is 

/>oo 

J(xq,u) = / c(x(t;xo,u),u(t)) dt, 
Jo 

while the optimal value function is 

V(x) = m{{J(x,u) | u : [0, oo) — > X measurable}. 

The optimal value function is the solution of the Hamilton- J 'acobi- Bellman equation (see [Son98] Chap- 
ter 8) 

inf { W(x) • f(x, u) + c(x, u)} = 0, 

with the boundary condition V(0) = 0. From this equation, an optimal feedback can be constructed by 
choosing a minimizing u to a given x (assuming that it exists). 

We call a control system (1) with cost function c uncoupled if one can decompose the state space and 
the set of controls into at least two subsystems, X = ® i=1 Xi and U = ® i=1 Ui, k>2, such that / and 
c can be written as 

f(x,u) 

a 1 1 1 1 c(x, u) — 2i=i c i{ x %i u i)i where fi:XiXUi—> R ni , a : Xi x Ui — > [0, oo) and ni + . . . + = n. 
Of course, for an uncoupled system, the optimal value function is simply the sum of the optimal value 
functions of the subsystems: 

k 

V(x) = J2Vi(xi)- 

i=l 

Evidently, if V is C 2 , then 9i,jV = 0, but also the converse is true (for the proof, see the appendix): 
Proposition 1. For V G C 2 (X) the following statements are equivalent. 

a) There are Vi £ C 2 {Xi), i = 1, . . . , k, such that V(x) = 53j=i Vi( x i)- 

b) It holds that dijV = for all i,j € {1, .. . , k}, i ^ j. 

Accordingly, a control system / is called weakly coupled if again there is a decomposition of X and 11 
as above such that for all i,j 6 {1,..., k}, i ^ j, 

U^x.u)!! » ||%A(a;,u)|| 

uniformly in x and u, for a given norm || • || (on the Xi). We define the coupling constant as 

\\djfi(x,u)\\ 

e f := max sup — —. 

i^j x.u \\difi(x,u)\\ 



fi(xi,u±) 
fk{xk,u k ) 
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Correspondingly, a system is weakly coupled if £/ <C 1. Note that this notion is rather vague — in the 
sequel we will make statements about the asymptotic case Ef — > 0. 

In light of Proposition 1 a natural question is whether for weakly coupled systems the mixed derivatives 
dijV for the optimal value function are uniformly small. 

3 Weakly coupled LQ systems with linear feedback — continu- 
ous time 

We now consider more specifically linear time invariant systems with quadratic cost function. A weakly 
coupled linear system is defined by 

x = Ax + Bu, (2) 

where A is a weakly coupled matrix, i.e. the diagonal blocks dominate the other ones (an implication of 
the definition above), and B is a block diagonal matrix (i.e. the subsystems are controlled separately). 
The accumulated cost along a trajectory x(t) = x(t; xq, u(-)) is given by 

/>oo 

J(x ,u(-))= / x(t) T Qx(t) +u(t) T Ru(t)dt, 
Jo 

where Q and R are symmetric positive definite (spd) matrices, both block diagonal to satisfy c(x,u) = 
J3i=i C-iixi, Ui). The optimal value function is then the infimum over all possible control functions. 
Imposing no restrictions on u, the unique optimal control is realized by the feedback 

u(x) = -R~ 1 B T Px, 

where P is the unique spd solution of the Riccati equation 

PBR- 1 B T P - PA- A T P - Q = 0, (3) 

and the optimal value function is given by V(x) — x T Px; see [Son98], chapter 8.4. 

Hence, the question whether the dijV are small, reduces to the question whether the off-diagonal 
blocks of P are small compared to the diagonal ones. Our answer will be of asymptotic nature, i.e. by 
considering small perturbations of an initially block diagonal (uncoupled) system matrix A. Note, that 
for a block diagonal system matrix and an analogously block diagonal ansatz P equation (3) simplifies 
to k uncoupled Riccati equations. Then, by uniqueness of the solution is P block diagonal as well. 

In order to analyze the effect of perturbations on A, we define the separation of two matrices; see 
e.g. [Var79]. 

Definition 2 (Separation of matrices). The separation of two square matrices X and Y, sep(X, Y), is 
defined as the smallest singular value of I <g> X — Y T ® /, where I is the identity and <g> denotes the 
Kronecker product. 1 

The following estimate plays a central role in our considerations. Although more general results 
exist [KPC86], a proof of this one is also given, because we use the ideas in it again later. 

Theorem 3. Let P(A) denote the solution of (3) for fixed B, Q and R, in dependence of A. If P(A + 
5 A) = P(A) + SP, and A c \ — A — BR _1 B T P(A) denotes the closed-loop system matrix, it holds that 

x It holds that sep(X,Y) = sep(Y,X), and sep(X T ,Y T ) = sep(X, Y). 
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Proof. Define the function g(A, P) = PBR~ 1 B T P — PA — A T P — Q, where P is assumed to be a sym- 
metric matrix. By definition, g(A, P(A)) = for all A. We have that P(-) is real-analytic in an appro- 
priate neighborhood of A [Del83] . It also holds 

d A g(A,P)-X = -PX-X T P 

d P g(A,P)-X = XBR- 1 B T P + PBR- 1 B T X -XA-A T X. 

From the implicit function theorem we have 

DP(A) = -d P g (A, P (A))- 1 d A g(A, P(A)), 

if dp g (A, P (A)) is invertible. Following [Var79], one can see that this is the case if and only if 
sep(yl c (, — A^j) 7^ (in fact, dpg(A, P(A)) is a linear operator with matrix representation — I®A c i — A c i®I 
if X is represented as a long vector). Moreover, it holds that 

\\DP(A)\\ F < ^|fi%r - (5) 

A Taylor expansion of P in A yields the claim. □ 

An immediate consequence of Theorem 3 for our situation is that if we are given a weakly coupled 
linear system (2) and we can decompose A = A + SA such that ||<^4|| F <C ||-Ao|| F , and ||_DP(A )|| F is 
small, then the optimal value function is "weakly coupled" , i.e. its mixed second derivatives are uniformly 
small. 

Remark 4. 

a) Note that (5) is an a posteriori estimate, i.e. it does not characterize the sensitivity directly in terms 
of A, B, Q and R, but involves the solution P of the (uncoupled) Riccati equation. 

b) In [KPC86] (see also [KGMP03]) the authors present a non-local perturbation analysis of the Riccati 
equation (3). This could be used to yield better bounds for the estimates of the mixed second derivatives 
of the optimal value function. With the present result we have no control about the quality of our 
estimate, since we have no information about the magnitude of higher order terms. 

Their result states that \\5P\\ F < #(||<L4|| F ) for \\5A\\ F G [0,a*), with 



g(a) = Y d ( s ~ 2a ~ (( s " 2a ) 2 - 8 P da ) 



where s = sep (Ad, —A^j, d — \\BR ^^Ip and p = ||P(A)|| F . Here, the right boundary point a* is 
defined as the (smaller) positive root of 2a + 2^/2dpa — s = 0. 



4 Coupling-adapted estimates 

A somewhat unsatisfactory property of the Frobenius norm ||-|| F is that ||<L4.|| F is not closely related to 
the coupling constant of the system. Much more adequate for measuring the coupling of a system is the 
following norm, which focuses on the subsystems. Each matrix X £ W ixn , with n — 53i=i ri ii can ^ e 
partitioned blockwise, according to the subsystems: X = (l,j)- J=1 , where X t j G R niX,lj . Define 

ll^llc = max ll^jllF • 
«.j 

Note that this norm is not sub- multiplicative, but it holds that 

k 



AB\\ C = max A^Bij < k \\A\\ C \\B nc . 

1 '3 yl i F 



1=1 



Adapting the proof of Theorem 3 to an uncoupled system, we obtain the following sensitivity estimate: 
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Lemma 5. Consider the situation of Theorem 3 and suppose that the system is uncoupled. Then 

\\(DP(A) ■ SA) it j\\ ¥ < ll^)MllF + ll^)gll F max | ¥Ai< ,|| f ; n^ii } . (6) 



Note that this block-wise sensitivity estimate directly implies an analogous estimate in the norm 



\c- 



Proof. Note, that the system is uncoupled, hence A, P{A) and A c \ are all block-diagonal. It holds that 
block-wise 

(d A g(A,P) ■ 8A)ij = -Pi,iSA itj - SAj.Pj. 



Recall that dp g(A,P) ■ X = —XA c i — A^X =: —S for symmetric matrices X. It holds that block-wise 



3 1 l,l"^H,J "^j,^ 0,3- 

Xij(A c i)jj + (Ac^f^Xij = Sij = —Pi^SAij — SAj^Pjj. 

Solving this for j implies the claim analogously to the proof of Theorem 3. □ 

The previous result also gives us a possibility to measure the effect on the coupling constant of the 
closed-loop system. 

Corollary 6. In the situation of Lemma 5 the first order perturbation 5A c i of the closed loop system 
matrix induced by the perturbation 5 A of the system matrix A satisfies 

\ $eP{{Acl)j,j, -K A el)i,i) J 

(where < means that this inequality is satisfied up to higher order terms). 

Proof. Use A cl = A - BRT 1 B T P(A). □ 

5 Weakly coupled LQ systems with linear feedback — discrete 
time 

The considerations here are analogous to the ones in sections 3 and 4. The discrete time control system 
is given by 

x m+ x = Ax m + Bu m , m = 0,1,2, 

We assume A to be weakly coupled and B to be a block diagonal matrix. The cost accumulated along a 
trajectory (x m ) meN = {x m (x ; (u fc ) fceN )) me N is given by 



J(x , (u k )ken) = xJ n Qx m + uJ n Ru m , 



m=0 



where Q and R are block diagonal spd matrices. If there are no restrictions on u the optimal feedback is 
given by 

u{x) =-(R+ B T PBy 1 B T PAx, 

and the optimal value function is V(x) = x T Px, with P being the unique spd solution of the discrete 
Riccati equation (see [Son98], chapter 8.4) 

P = A T (p - PB (R+ B T PBy 1 B T P^j A + Q. (8) 

In the following we derive a first order perturbation result on P depending on A. 
For arbitrary square matrices M±, Mi and 5* the equation 

X - M?XM 2 = S 
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is solvable if the matrix / — <£> is nonsingular, and the absolute condition (w.r.t. the Frobenius 
norm) of the solution is given by l/sep#(M) with 

sep # (Mi, M a ) := (r mia (I - M[ ® M 2 T ), 

where cr m i n (-) denotes the smallest singular value. 

We show directly the block-wise sensitivity estimate. 

Theorem 7. Let the linear discrete time system be uncoupled, and let P(A) denote the solution of (8), 
where B, Q and R are fixed; and let A c i denote the closed-loop matrix, i.e. A c i = A — B(R + B T PB)~ 1 B T PA 
. Then for some perturbation 5 A of A we have 

\\Pi i(A c i)i i\L + ||(A oi )T,-A 5 'L 

\\(DP(A) ■ 5A)ij\\ F < Infh \a < > ^ ,J " F nmx{pA M || F ,||M J/t || F }. (9) 

Proof. Just as in the proof of Theorem 3, we use the implicit function theorem. For this, let 
g*(A, P) = A T (PB (R + B T PBy 1 B T P -P^jA + P-Q. 

Setting A = A - B(R + B T PB)- 1 B T PA, we obtain 

d A g* (A, P) ■ X = -X T PA - A T PX 
d P g*{A,P)-X = X-A T XA. 

Since all matrices involved are block diagonal, we have in particular the block wise equations 

(d A g*(A, P) ■ X). t . = -Xj^jAj,; - Al-P^X^ 
(d P g*(A,P)-X). t . = Xu - AjjXijAjj. 

Since A = A c i for P = P(A), considerations as in the proof of Theorem 3 and Lemma 5 imply the 
claim. □ 

Remark 8. For the discrete time case as well, in [KPC86] the authors present a non-local perturbation 
analysis in the Frobenius norm for the solution of (8). 

6 Coupling estimates in dependence on the number of subsys- 
tems 

In this section we compare the sensitivity estimates obtained in the Frobenius and coupling-adapted 
norms. The exposition is restricted to the continuous-time case, but analogous results hold in discrete 
time as well. 

The estimates (5) and (6) are easier to compare if we see, that the denominators are the same: 
Proposition 9. For a block-diagonal matrix we have 

sep(A, — A T ) = minsepfAj — AjA. 

i,j ' J ' J 

Proof. For block-diagonal A the Lyapunov equation A T X + XA = Q is decoupled, 

AfjXij + XjAjj = Qij. 

This means that the solution operator of the Lyapunov equation is block-wise decoupled, hence its singular 
values are the singular values of the blocks. This proves the claim. □ 
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Consider now the coupling of k identical subsystems, i.e. A — Ikxk <8> Ao, B = Ikxk <& Bo, etc. Then 
P = Ikxk ® Pqi where Po solves the Riccati equation with Aq, Bq, etc. In particular, the denominator 
in (5) stays constant in the number of subsystems k. Hence, the right hand side of the Frobenius norm 
estimate increases as k 1 ^ 2 . Using (5) to obtain an estimate in we can use norm equivalence. Having 
k subsystems, one verifies easily that 

\\A\\ c <\\A\\ F <k\\A\\ c . 

Thus, we have 

\\SP\\ C < \\SP\\ F < \\DP(A)\\ F \\SA\\ F < k \\DP(A)\\ F \\SA\\ C . 

In general, this would imply an estimate ~ fc 3 / 2 . 

However, the coupling-adapted norm estimate (6) shows that the first order perturbations do not 
depend on the number of subsystems at all. An important conclusion is that the coupling strength of the 
optimal value function does not increase with the number of subsystems. 

Remark 10. This comparison gets important if we would try to devise an approximation technique 
for the OVF. The estimates suggest, that we should search for an approximation with error bounded by 
some analog of the the coupling- adapted norm for functions, because it does not scale with the number of 
subsystems. Based on our former considerations, in particular Proposition 1, an ideal candidate for the 
error indicator would be max^j ||<3» jV||oo. 



7 The optimal value function of destabilized systems 
7.1 Continuous time 

In this section we investigate the structure of the optimal value function if a coupling destabilizes the 
optimally controlled (linear time-continuous) system. 

The norm of the smallest perturbation 5 A that destabilizes A, the so-called stability radius of A [HP86] , 
is given by 

r(A):= min { \\8A\U I 3z G a(A + SA), $lz = 0) . 
SAeC nXn ' 

So if SA destabilizes A, then ||<L4||2 > r (A). What can be said about the estimate (4) in these cases? 
Proposition 11. It holds that 

sep(A,-A T ) < 2r(A). (10) 

This result seems to be first stated in [KH90] Lemma 2.7; see also the remark after Corollary 2.5 in 
the same paper. They refer to Theorem 2.2 in [HK88] for the proof. If the stability radius is measured 
by the Frobenius norm instead of the spectral norm, the same statement can already be found in [Loa85] . 
His result follows from the aforementioned ones by simple norm estimates. Since our short proof differs 
significantly from the ones used in these papers, we include it in the appendix. 

Proposition 11 together with (4), shows that if a weak coupling (i.e. with a small constant e > 0) 
destabilizes a system, we have to take a strong coupling (i.e. with constant of size 0(1)) of the optimal 
value function into account. To see this, note that for a destabilizing coupling it holds that ||<L4||2 > r(A c i), 
hence we have in leading order 

\\SP\\ F 2\\SA\\ f 

\\P\\f - sep(A cl ,-AT)- ( > 

V v ' 

>1 

But does the coupling of the OVF stay small if r(A c i) is not small? An answer can be given using the 
estimate of He [He97], Theorem 4; 

irr{A cl ) 2 \\A cl \\ 2 . T , ri9 s 
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Assume that r(A c i) > g\\A c i\\ 2 , with some robustness constant g e (0,1). Combining the estimates (4) 
and (12) yields in leading order 



\\sp\\ f K 2 {■?+*) 



\\P\\f ~ n\\A cl \\ 2 

Thus, if the robustness constant g is of order one, we expect a small coupling to induce also only a small 
coupling in the OVF (assuming that "small coupling" means ||<5j4||f "C H^-cilh here). 

Remark 12. The previous bound suggests, that in case of g <C 1 it might be possible that a perturbation 
\\SA\\f <C r(A c i) induces large changes in the OVF, since \\6A\\f/q 2 can be big. Unfortunately, we did 
not find any examples supporting this claim. 

The above bounds did not consider whether the perturbation of the system matrix comes from a 
coupling or not. We would like to consider this topic in the following. 

Corollary 13. For a block diagonal matrix A we have 

sep(A, — A T ) = min sep(Ai i,—Aj A < 2mmr(An). 

This could be interpreted as "a system with robust subsystems yields a robust OVF"; however a 
formal proof would involve an estimate analogous to (12). We do not pursue this question here. Instead, 
we would like to characterize the sensitivity of the OVF in terms of the stability radius with respect to 
the coupling adapted norm || • \\c- Define 

r c (A) := ^nrin^ {||<L4|| C | 3z e cj(A + 6A),$tz = 0} , 

and note ||A||2 < k \\ A\\ c . It follows sep(A, — A T ) < 2kr (A). If we restrict the class of allowed perturba- 
tions to the ones with zero diagonal blocks (i.e. "coupling induced perturbations"), we immediately get 
the slight improvement sep(A, —A T ) < 2(k — l)r c (A). With this bound, the analogous estimate to (11) 
would be 

\\6P\\ C 2\\5A\\ C 

\\P\\c ~ S ep(A ch -Ary [ > 

s V ' 

which does not suggest the intuition of a strongly changing OVF under a weak destabilizing coupling 
provided k, the number of subsystems, is large. It would be desirable to find more sophisticated bounds of 
the separation in terms of the stability radius, in particular considering structured stability radii (adapted 
to perturbations induced by the coupling) , in the sense of [KHP06] . 

Remark 14. It should be noted that other characterizations of the OVF than (3) exist, and hence other 
sensitivity estimates than (5) could be obtained; see [Meh91, KGMP03]. 



Examples. We consider two different types of examples. First, where the system matrix is strongly 
non-normal, so the pseudospectra [TE05] corresponding to small perturbations reach into the positive 
complex half plane. Second, where the control part of the cost (i.e. the matrix R) is much bigger than 
the state part of the cost (i.e. the matrix Q), hence the optimally controlled system is weakly stable in 
the sense that the spectral abscissa of the closed-loop system matrix is small (compared to the modulus 
of the corresponding eigenvalues). In all examples we take R to be the identity matrix. At the end of 
each example sep(A c i, — A^) and r(A c i) are shown. 

Example 15. The optimally controlled system has robust stable eigenvalues, i.e. the spectral abscissa is 
of order 0(1). A weak coupling destabilizes the closed-loop system. 



/l 1000 
A=\0 -0.5 
\o -1 









)■ M 









Vo 
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Closed-loop eigenvalues: — 0.5, — \/2, — \/2. The coupling matrix 



5A = 




-0.1 
>0.04 



moves the eigenvalues to — 1.6707 ± 0.8163i, 0.0130, and (with two digits precision) 

5P + P = 



2.4 130 
130 93000 




0.62 120 -1.3 
120 33000 -450 
-1.3 -450 8.5 



sep(A cl , -A*) = 4.0 • 10- 5 . r(A cl ) = 2.8 • IO" 3 . 



Example 16. The optimally controlled system has robust stable eigenvalues. A weak coupling destabilizes 
the closed-loop system. 





(I 


50 








\ 




- 


-0.5 


50 








A = 








-0.5 


50 
















-0.5 







\o 











0.5/ 


eigenvalues 


are 


-1.11, 


-1.41, 


-0.5, 



B 



(l 0\ 







\o 1/ 



Q 



'5x5- 



SA 



( o 







\ 





0.1 



\0.001 / 

moves the eigenvalues to 0.39 ± lAi, -1.9 ± 0.38i, -1.1. sep(A cl , -A^) = 4.4- IO" 11 . r(A cl ) = 2.2- 10~ 6 . 

Example 17. The optimally controlled system is only weakly stable due to an expensive control. A weak 
coupling destabilizes the closed-loop system 



A = 



/io- 2 


i 





^ 


-1 


io- 2 














io- 2 


1 


V o 





-1 


io- 2 y 



B = 



(1 0\ 



1 

\0 OJ 



Q = 10- i h 



The closed-loop eigenvalues are —0.012 ± li, —0.012 ± li. The coupling matrix 

( 

5 A = 



5-10~ 2 0\ 



2 • 10~ 2 

\ 0/ 



moves the eigenvalues to — 0.028 ± li, 0.0036 ± li. The matrices of the associated optimal value functions 
are 



10- 



( 4.4 
-0.05 


V o 



-0.05 
4.4 








4.4 
-0.05 



\ 


-0.05 
4.4 / 



P + SP = 10" 



/ 7.6 -0.08 4.6 -0.05\ 

-0.08 7.6 -0.04 4.6 

4.6 -0.04 3.8 -0.04 

\-0.05 4.6 -0.04 3.8 / 



Clearly, the off-diagonal blocks of P+5P are not small in comparison to the diagonal blocks. sep(A c ;, —A^) 
0.025. r(A cl ) = 0.092. 
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7.2 Discrete time 

Just as before, we compute a bound for the sep#(-, •) term in (9). Recall 

sep # (Mi , M 2 ) = min II X - MTXM 2 \\ f . 
pf||,=i" 

We would like to bound sep#(A, A) in terms of the stability radius of A, defined by 



We have that 
Proposition 18. 



r(A):= min { \\5A\\ 2 I 3A € C : |A| = 1, A € a(A + 5A)\ 



sep*(A,A) <r(A) (2 + r(A)). 



Just as in the continuous-time case, this result, together with (9), shows that if a weak coupling (i.e. 
with a small constant e > 0) destabilizes a system, we have to take a strong coupling (i.e. with constant 
of size 0(1)) of the optimal value function into account. 

7.3 A discrete-time non-linear example 

The following system describes the temperature evolution of a fluid in two coupled reactors. It is a sim- 
plified version of [SS11], and the constants may vary as well. The system consists of two one-dimensional 
subsystems: 

*i = 777^ ( 2 - 133 ' icr6fc 2 (x 2 - £i) + g c (wi)(285.65 - xi) + 8.535 • 1(T 4 ) 

0.0261 ( 14 ) 

x 2 = (1.795 • 10- 5 (293.15 - x 2 ) + 177.6 • W^k^ - x 2 ) + 7.622 • 10- 5 u 2 ) 

where the parameters k\ = 0.1 and k 2 = 0.2 represent the coupling strength, ui,u 2 € [0, 1] are control 
parameters, and 

/ -(«i-0.15)\ 

83-(l-e — —i ) for m> 0.15 



other wise. 



q c {ui) = 1 ■ 10- 6 

The target set, where the system should be controlled to, is not a point, but a set, namely [295.03, 299.71] x [304.40, 308.15]; 
and the cost functions are 

ci(xi, u x ) = 0.01 • \x x - 297.37| + (u x - ui) 2 , c 2 (x 2l u 2 ) = 0.01 • \x 2 - 306. 28| + {u 2 - u 2 ) 2 , 

where Ux, 2 are chosen such that they hold the respective subsystem in the middle of the target (if no 
coupling is present). The state space is X = [285.65,323.15] x [293.15,323.15], and the control space 
is U = [0,1] x [0,1]. 

The time-T-map of (14) with T = 200 will be the discrete-time system under consideration. We 
applied the method developed in [GJ08] to compute the OVF in the non-coupled, and coupled cases. 
Figure 1 shows the results of the computation. Despite the weak coupling (the coupling constant is 
« 0.1), the OVF changes significantly: the relative difference of the coupled OVF to the non-coupled is 
more than 100% in the L 2 norm, and more than 80% in the maximum norm. This is not surprising, since 
the coupling is chosen so that it de-stabilizes the optimally controlled non-coupled system. However, it 
is interesting to note that the best approximation of the coupled OVF in the L 2 norm by functions of 
the form V\(x{) + V 2 (x 2 ) yields a relative error of only 16%! While the OVF changes significantly due to 
the weak coupling, it's approximability by non-coupled functions seems to be kept. 
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290 295 300 305 310 315 320 290 295 300 305 310 315 320 



Figure 1: The OVF of the non-coupled (left) and the coupled (right) system. The white rectangle 
indicates the target set. 

8 Conclusions 

We have shown that even if one can divide an optimal control problem into subsystems which appear 
to be weakly coupled according to the structure of the Jacobian, the optimal value function still might 
not inherit this structure. We have also made an attempt to find a suitable norm for describing the 
sensitivity of the coupling strength of the optimal value function with respect to weak coupling in the 
system matrix. This "coupling adapted" norm turned out to be advantageous for upper bound estimates 
(cf. Section 6), but does not seem to give a good intuition on the large sensitivity in the case of a large 
number of subsystems and a destabilizing coupling (cf. Section 7). A better understanding is needed of 
how the coupling of the system matrix transfers into the coupling of the optimal value function. Certainly, 
more sophisticated ways of defining and detecting "weak coupling" are imaginable which would not show 
this undesired effect. Nevertheless, we have shown that the choice of coupling measure already has a 
significant effect on the obtained sensitivity bound. 

On the way to designing (nearly) optimal controllers to weakly coupled systems a natural next question 
would be how does the optimal value function look like if additional robustness constrains (e.g. r(A c i) 
has to be greater than a given threshold) have to be met by the closed loop system (if an optimal value 
function exists under such constraints at all); how it can be computed, and how does it compare to the 
optimal value function of the non-coupled system. Also, it might be possible to define cost functions such 
that the feedback for the associated uncoupled system still steers the weakly coupled system into some 
neighborhood of the target set. These are questions for further investigations. 

9 Appendix 

Proof of Proposition 1: 

Proof. "=►": Trivial. 

"-4=" : In order to show the claim, we need following sublemma: 

Sublemma: If V is C 2 in the X\ and x 2 variables (let z denote the other variables), and di^V(xi, x%, z) = 0, 
then V(x%,X2, z) — Vi(xi,z) + V2(x2,z), where V% is C 2 in the Xi variable. 
Proof of sublemma: It holds that 

/■X2 

d\V{x\, X2, z) — 9iV(a;i, 0, z) = / di^Vfai, s, z) ds = 0. 

J o 

Again by the fundamental theorem of calculus we have 

V(x 1 ,x 2 , z) - V(0, x 2 , z) - V{xi, 0, z) + V(0, 0, z) = 0, 
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i.e. V(xi,X2,z) — Vi(xi,z) + V~2(x2,z). From V\ = V — V%, and d\V% = 0, the rest of the claim follows 
immediately. 

The proof of the proposition follows by induction. For k — 2 the sublemma implies the statement. For 
the induction step, assume that 

V( x ) = Si=i V%i x ii z )i where z = {xt+i, . . . , Xk) and the Vi are all C 2 . It follows for all i 

= d it i + iV(x) = d it £ + iVi(xi, Xi + i, z'), 

where z' = {xi+2, ■ ■ ■ , %k)- The sublemma implies 

Vi(xi,x e+1 ,z') = VS 1 ' (x i} J) + vy\x i+1 ,z'), 

and V t {2) being C 2 in Xi and xi+i, respectively. Hence we have V(x) = Yli=i Wi(xi,z'), where 

Wi( Xi ,z') = V^\ Xi ,z'), i= !,...,£, 
i 

W e+1 (x e+1 ,z r ) = J2 V f\ x W> z ')- 

3=1 

This completes the proof. □ 
Proof of Proposition 11: 

Proof. We make use of the notion of pseudospectra [TE05]. The set 

a E (A) := {z eC\35AeC nxn : \\6A\\ 2 < e, z £ <j(A + 8A)} 

is the (complex) e-pseudospectrum, where cr(-) denotes the usual spectrum. 
From Theorem 3.1 [Var79] we have 



sep(A,-A T ) < sep x (A,-A T ) := min{e 1 +e 2 \a ei (A)na e2 (-A T ) ^ 
< min {ei + e 2 | 3z £ iR, z £ a El (A),z £ a E2 (-A T )} 



ei, £2 

= 2r(A). 

The inequality in the second line comes from restricting the minimization to the imaginary axis; the 
equation in the third line comes from the fact that iu £ a {A + 5 A) for some ui £ E implies iuj £ 
o-(—A T — 5 A*), where A* denotes the conjugate transpose of A; and that ||A||2 = ||^4*||2- n 

Proof of Proposition 18: 

Proof. Assume we have A £ C, |A| = 1, non-negative numbers £i t 2, vectors || f 1,2 1| 2 = 1 and ||u>i,2||2 = £1,2 
such that for the real matrices M\ t 2 holds 

{M? - XI)vi = Wi, i = l,2. 

It follows (x* denotes the conjugate transpose of x) 

Wl w* 2 = (M X T - XI)viv^(M 2 - XI) 

= M^ v\v 2 M 2 — Xv\v\M% — XM/[ v\v* 2 + viv 2 

= M^ v\v 2 M 2 — Xv\{Xv\ + 11%) — X(Xv\ + W\)v\ + v\v 2 

= All \\v 2 M 2 — v\v 2 — Xviw 2 — Xu>iv 2 . 

With this equation and by setting X = V\v\ we conclude that sep # (Mi, M 2 ) < £162 + £1 + £2- 

Since a A £ C is in the (complex) e-pseudospectrum of A if there is a vector v of unit modulus such 

that || (A - A7>|| 2 = e (see [TE05]), we can find a ||u|| 2 = 1 such that \\{A - XI)v\\ 2 = r{A). 

Setting vi = V2 = v and Mi 2 = A in the above computation yields the claim. □ 
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